rustc_parse: Remove `Parser::normalized(_prev)_token` #69801

petrochenkov · 2020-03-07T14:03:25Z

Perform the "normalization" (renamed to "uninterpolation") on the fly when necessary.

The final part of #69579 #69384 #69376 #69211 #69034 #69006.
r? @Centril

petrochenkov · 2020-03-07T14:03:53Z

bors · 2020-03-07T14:04:08Z

⌛ Trying commit f572d400edc7f52ce6597f359b291dc1322de3c7 with merge ac2bf9054006fee0f39772b8daad9845a9fa570f...

bors · 2020-03-07T16:58:00Z

☀️ Try build successful - checks-azure
Build commit: ac2bf9054006fee0f39772b8daad9845a9fa570f (ac2bf9054006fee0f39772b8daad9845a9fa570f)

petrochenkov · 2020-03-07T16:59:27Z

@rust-timer queue

rust-timer · 2020-03-07T16:59:28Z

Awaiting bors try build completion

petrochenkov · 2020-03-07T17:04:19Z

@rust-timer build ac2bf9054006fee0f39772b8daad9845a9fa570f

rust-timer · 2020-03-07T17:04:21Z

Queued ac2bf9054006fee0f39772b8daad9845a9fa570f with parent a039217, future comparison URL.

rust-timer · 2020-03-08T09:01:14Z

Finished benchmarking try commit ac2bf9054006fee0f39772b8daad9845a9fa570f, comparison URL.

Centril · 2020-03-08T09:11:42Z

Haven't reviewed the PR yet, but those numbers seem like a slight regression overall, although with some improvements?

petrochenkov · 2020-03-08T10:28:26Z

I expected the perf diff to be within the noise.
Let's check whether it's reproducible.

@bors try @rust-timer queue

rust-timer · 2020-03-08T10:28:28Z

Awaiting bors try build completion

bors · 2020-03-08T10:28:38Z

⌛ Trying commit bfc7502 with merge 5f66c2a...

@Centril

rustc_parse: Remove `Parser::normalized(_prev)_token` Perform the "normalization" (renamed to "uninterpolation") on the fly when necessary. The final part of #69579 #69384 #69376 #69211 #69034 #69006. r? @Centril

bors · 2020-03-08T13:28:15Z

☀️ Try build successful - checks-azure
Build commit: 5f66c2a (5f66c2afe2e46c53471539710934a5c1ab1d4ce5)

rust-timer · 2020-03-08T13:28:17Z

Queued 5f66c2a with parent f943349, future comparison URL.

rust-timer · 2020-03-08T19:25:28Z

Finished benchmarking try commit 5f66c2a, comparison URL.

petrochenkov · 2020-03-08T19:28:23Z

Yeah, the perf diff is noise.

Centril

r=me with these comments considered.

src/librustc_expand/mbe/macro_parser.rs

src/librustc_parse/parser/expr.rs

src/librustc_parse/parser/item.rs

src/librustc_ast/token.rs

Centril · 2020-03-09T07:57:29Z

src/librustc_ast/token.rs

@@ -714,6 +726,24 @@ pub enum Nonterminal {
 #[cfg(target_arch = "x86_64")]
 rustc_data_structures::static_assert_size!(Nonterminal, 40);

+impl Nonterminal {
+    fn span(&self) -> Span {


Maybe we should add trait HasSpan { ... } for this. I've felt the need for it in reviewing / writing other PRs as well.

Maybe.
It's not too helpful for this PR though because all the uses of spans are concrete.
I'm not sure where exactly it would be useful, actually.

Mainly where you have slices of objects with spans, and you want to have "first to last"; it seems to occur sometimes. I agree though that for the purposes of this PR it doesn't add much, so let's defer this to some other PR. :)

src/librustc_parse/parser/mod.rs

petrochenkov · 2020-03-09T10:09:01Z

Updated.

Centril · 2020-03-09T10:13:10Z

@bors r+

bors · 2020-03-09T10:13:11Z

📌 Commit 7a30bb1 has been approved by Centril

bors · 2020-03-09T10:13:12Z

🌲 The tree is currently closed for pull requests below priority 1000, this pull request will be tested once the tree is reopened

@ghost

Rollup of 6 pull requests Successful merges: - #69201 (Permit attributes on 'if' expressions) - #69685 (unix: Don't override existing SIGSEGV/BUS handlers) - #69762 (Ensure that validity only raises validity errors) - #69779 (librustc_codegen_llvm: Use slices in preference to 0-terminated strings) - #69801 (rustc_parse: Remove `Parser::normalized(_prev)_token`) - #69842 (Add more regression tests) Failed merges: r? @ghost

@Centril

parse: Tweak the function parameter edition check Follow-up to rust-lang#69801. Edition of a code fragment is inferred from "the place where the code is written". For individual tokens like edition-specific keywords it may be the span of the token itself ("uninterpolated" span), but for larger code fragments it's probably not, in the test example the trait method is obviously written in "2015 edition code". r? @Centril

@Centril

parse: Tweak the function parameter edition check Follow-up to rust-lang#69801. Edition of a code fragment is inferred from "the place where the code is written". For individual tokens like edition-specific keywords it may be the span of the token itself ("uninterpolated" span), but for larger code fragments it's probably not, in the test example the trait method is obviously written in "2015 edition code". r? @Centril

rust-highfive assigned Centril Mar 7, 2020

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Mar 7, 2020

petrochenkov added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Mar 7, 2020

petrochenkov force-pushed the nonorm branch from f572d40 to bfc7502 Compare March 8, 2020 10:26

petrochenkov removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Mar 8, 2020

Centril reviewed Mar 9, 2020

View reviewed changes

petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Mar 9, 2020

petrochenkov added 4 commits March 9, 2020 12:42

rustc_parse: Use Token::ident where possible

925e9a2

rustc_ast: Introduce Token::uninterpolated_span

f4a03c4

rustc_ast: Introduce Token::uninterpolate

43b27df

rustc_parse: Remove Parser::normalized(_prev)_token

5d7f67d

petrochenkov added 2 commits March 9, 2020 12:42

Use Token::uninterpolate in couple more places matching on (Nt)Ident

9be233c

Address review comments

7a30bb1

petrochenkov force-pushed the nonorm branch from bfc7502 to 7a30bb1 Compare March 9, 2020 10:08

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Mar 9, 2020

Centril mentioned this pull request Mar 9, 2020

Rollup of 6 pull requests #69851

Merged

bors merged commit 2677d59 into rust-lang:master Mar 9, 2020

petrochenkov mentioned this pull request Mar 10, 2020

parse: Tweak the function parameter edition check #69896

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rustc_parse: Remove `Parser::normalized(_prev)_token` #69801

rustc_parse: Remove `Parser::normalized(_prev)_token` #69801

petrochenkov commented Mar 7, 2020

petrochenkov commented Mar 7, 2020

bors commented Mar 7, 2020

bors commented Mar 7, 2020

petrochenkov commented Mar 7, 2020

rust-timer commented Mar 7, 2020

petrochenkov commented Mar 7, 2020

rust-timer commented Mar 7, 2020

rust-timer commented Mar 8, 2020

Centril commented Mar 8, 2020

petrochenkov commented Mar 8, 2020

rust-timer commented Mar 8, 2020

bors commented Mar 8, 2020

bors commented Mar 8, 2020

rust-timer commented Mar 8, 2020

rust-timer commented Mar 8, 2020

petrochenkov commented Mar 8, 2020

Centril left a comment

Centril Mar 9, 2020

petrochenkov Mar 9, 2020

Centril Mar 9, 2020

petrochenkov commented Mar 9, 2020

Centril commented Mar 9, 2020

bors commented Mar 9, 2020

bors commented Mar 9, 2020

rustc_parse: Remove Parser::normalized(_prev)_token #69801

rustc_parse: Remove Parser::normalized(_prev)_token #69801

Conversation

petrochenkov commented Mar 7, 2020

petrochenkov commented Mar 7, 2020

bors commented Mar 7, 2020

bors commented Mar 7, 2020

petrochenkov commented Mar 7, 2020

rust-timer commented Mar 7, 2020

petrochenkov commented Mar 7, 2020

rust-timer commented Mar 7, 2020

rust-timer commented Mar 8, 2020

Centril commented Mar 8, 2020

petrochenkov commented Mar 8, 2020

rust-timer commented Mar 8, 2020

bors commented Mar 8, 2020

bors commented Mar 8, 2020

rust-timer commented Mar 8, 2020

rust-timer commented Mar 8, 2020

petrochenkov commented Mar 8, 2020

Centril left a comment

Choose a reason for hiding this comment

Centril Mar 9, 2020

Choose a reason for hiding this comment

petrochenkov Mar 9, 2020

Choose a reason for hiding this comment

Centril Mar 9, 2020

Choose a reason for hiding this comment

petrochenkov commented Mar 9, 2020

Centril commented Mar 9, 2020

bors commented Mar 9, 2020

bors commented Mar 9, 2020

rustc_parse: Remove `Parser::normalized(_prev)_token` #69801

rustc_parse: Remove `Parser::normalized(_prev)_token` #69801